Service- Learning Evaluation Projects 
A Step-by-Step Guide 

KEVIN RILEY 

Combining service-learning with an evaluation project offers 
maximum benefits to students and agencies alike. 

A s individuals prepare for their future as recreation management professionals, 
it is imperative that they receive opportunities to put theory into practice. 
Service-learning projects fulfil I thisneed because they combine specific peda- 
gogical goalswith practical community service. Researchershave suggested that 
service-learning isa great teaching tool that can complement and enhance traditional 
learning methods, such as public speaking, reading, and writing (Clark, 2002; Kunstler, 
2002). By participating in service-learning projects, studentsinternalizethetheoriesthey 
havelearned in courses, begin to think I ike professionals, and develop practical knowledge. 
Many recreation management programsacrossthenation have incorporated service-learn- 
ing projects into courses to enhance the development of future recreation professionals. 

In the recreation management program at Appalachian State University (ASU), ser- 
vice-learning projects have been implemented as part of the Evaluation of Recreation 
and Lei sure Servicescoursein order to providestudentswith opportunitiesto participate 
in the evaluation processes of local agencies. In rural North Carolina, many recreation 
practitioners and businesses lack the necessary resources to complete evaluation en- 
deavors because they are overburdened, understaffed, and underfunded. However, the 
evaluation process of recreation services, programs, participants, and facilities still must 
be accomplished. To assist with this dilemma, undergraduate students from ASU have 
been offering their services to local recreation and lei sure service providers for the past 
si x years. Th i s creates a servi ce-l earn i ng en vi ron men t th at al I ows both grou ps to ben efit. 



A student conducts a service-learning evaluation project at Appalachian Angler, a fly-fishing business in North Carolina. 
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For example, local recreational agenciesand businesses learn 
more about their clientele, programs, and policies, while 
ASU students learn by doing and make a better connection 
to evaluation theory. The purposeof thisarticleisto provide 
a description of an ASU evaluation service-learning project 
and the step-by-step approach used to design the project, 
and to outlinethebenefitsand challenges of implementing 
such a project. 

ASU Evaluation Service-Learning Projects 

In the evaluation course, students examine the methods, 
techniques, and application of evaluation in a variety of 
organizational functions that include clientele, programs, 
personnel, facilities, budgets, needs assessments, and eco- 
nomic impact estimations. Most of the evaluation projects 
thatthestudentshavecompleted for agencies have been very 
elementary; however, thequality of their efforts has caused 
many leisureserviceagenciesto request additional evaluation 
proj ects after the fi rst on e. Th e eval uati on proj ects are based 
on Henderson's (1995) trilogy of evaluation model, and on 
Rossman and Schlatter's (2000) program-planning model. 

The trilogy divides an evaluation endeavor into three 
specific components: (1) the criteria, (2) the evidence, and 
(3) the judgment. The criteria component is the overall 
framework for the eval uati on and is connected to the evalu- 
ation objectives. Theevidencesection addresses theproject's 
design, development of the survey, type of data collected, 
and how thedata will be analyzed. In thejudgment section 
the evaluators provide to the agency their conclusions and 
recommendations based on the data gathered from the 
survey. Most students and novice practitioners find that 


analyzing data is challenging. However, the program-plan- 
ning model focuses on simplicity and suggests that a good 
statistical analysis should organize data into a meaningful 
pattern that includes thefol lowing statistics: (1) distribution 
of scores(frequenciesand percentage), (2) measuresof central 
tendency (mean, mode, and median), and (3) measures of 
dispersion (standard deviation). Most students enrolled in 
thisclasswho master these ski I Is can report the outcomes of 
a program and interpret their meaning, all of which greatly 
assists local recreation agencies (Riley & Gaskill, 2004). 

Step-By-Step Approach to Program 
Implementation 

The following is a step-by-step account of how the evalua- 
tion service-learning project is integrated into the 15-week 
evaluation course at ASU (summarized in table 1). This 
approach is illustrated in the case study that describes one 
group's service-learning project (see sidebar, pp. 41-42). It is 
important to realize that in order for the project to benefit 
anyone the participating students must be prepared. This 
can occur only if the students take a synergistic approach to 
learning. This is accomplished by students attending class, 
reading required material, completing in -cl ass exercises, and 
com pi eti n g out-of-cl ass assi gn men ts. 

Step 1: Developing Evaluation Criteria 
and Objectives 

The first step of this process is essential to the effectiveness 
of the service-learning project. During the first week of the 
course, the instructor establishes the learning environment 

text continues on page 42 


Table 1. Steps to Implementing an Evaluation Service-Learning Project 


Step 1: Developing Evaluation Criteria and Objectives 

Week 1 Explain the service-learning project. 

Week 2 Establish evaluation teams. 

Week 3 Identify, contact, and negotiate evaluation objectives with selected agency. 

Week 4 Solidify evaluation criteria and objectives and provide to the course instructor written permission 

from the agency to conduct evaluation project. 

Step 2: The Proposal 

Week 5 Word-process the project proposal and submit to instructor for critique and approval. 

Upon approval, then submit to agency. 

Step 3: Survey Development 

Week 6 Develop the survey based on the objectives. 

Week 7 Instructor and peers critique survey. Submit survey to the agency for approval. 

Step 4: Data Collection 

Weeks 8-10 Collect data at the agency (multiple data collection days may be required). 

Weeks 11-13 Interpret data and create the evaluation report. 

Step 5: Presenting the Evaluation Project Report 

Week 14 Submit written report to instructor. 

Week 15 Professional presentation from each team is given to the class, instructor, and agency. 

Submit revised written report to the agency. 
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Case Study: TheAppalachian Angler Evaluation 

In thefol lowing sections, three studentsen rolled in the Evaluation of Recreation and LeisureServicescourse(P. Kraska, 

S. Slaton, andj. Spencer) explain their 2005 evaluation project, titled "Measuring the Appalachian Angler'sCustomer 
Satisfaction in Regard to the Retail Store, Guide Service, and the Effectiveness of Their Marketing Methods." 

Step 1: Developing the Criteria 

After we met as a group, we decided to work with Appalachian Angler (a fly-fishing company) for the service-learning 
project. Wethen metwith theowners of Appalachian Angler and developed an evaluation project that would measure 
the effectiveness of the company in terms of meeting its customers' needs. The owners welcomed our project, and 
after three meetings we developed the criteria and objectives to be evaluated for the study. Based on these conver- 
sations, we decided to investigate the following three areas: (1) effectiveness of the current web page, (2) customer 
satisfaction regarding the retail shop, and (3) customer satisfaction regarding the company's guide service. 

Step 2: The Proposal 

We submitted a written proposal that was reviewed and fine-tuned by our professor to ensure that the research ob- 
jectives would be measured correctly. In this written proposal, we reviewed the literature associated with the three 
evaluation objectives that would help us develop thesurvey. In addition, we created a time-line for collecting data 
and addressed how thesurvey would be administered to the Appalachian Angler customers. 

Step 3: Survey Development 

Our survey consisted of Likert responses, dichotomous statements, open-ended questions, and demographic ques- 
tions. The following specific objectives were created to pinpoint areas of concern: 

1. To measure whether products are reasonably priced. 

2. To measure whether customers feel that the store is well stocked to meet their needs. 

3. To evaluate staff performance. 

4. To assess how participants access information about Appalachian Angler. 

5. To measure how customers feel about Appalachian Angler's web site. 

6. To gauge whether customers want to see more on line features. 

7. To measure how well guides are transferring their knowledge to their customers. 

8. To assess whether guides are meeting customers at their skill level. 

9. To gather demographic information (name, address, phone numbers, email, age, and skill level) about the 
Appalachian Angler's customers. 

Objectives one through three assisted the owners of Appalachian Angler in determining whether they 
were meeting their customers' needs and whether improvements needed to be made to any of the specific areas. 
Objectives four through six evaluated the customers' preference for Internet marketing (e.g., the best way to reach 
the customer), possible new web page features (e.g., I inks to weather reports, fishing guides profiles, and sale items), 
and specific desired web media (e.g., compressed videos and tutorials). Objectives seven and eight evaluated the 
customers' satisfaction with thefishing guide's services. Objective nine gathered crucial marketing information for 
the company. 

Step 4: Data Collection 

The participants in this study were previous customers of Appalachian Angler. Theowners provided us with theemail 
ad d resses andtelephonenum bers of th ese cu stom ers. 

Data were obtained using three survey methods: online survey, paper-and-pencil survey (distributed in the retail 
store), and telephone survey. Originally, we felt that theon I inesurveys would be the most effective and efficient useof 
our time, si nee the customers' data would be captured in an Excel file and then analyzed. However, the on I inesurveys 
provided us with the lowest success rate, with only 15 percent of thecustomers providing information through this 
medium. We experienced problems with this method because some customers did not have the necessary software 
to complete the survey. Therefore, we decided to use a telephone survey, which gave us a return rate of 35 percent. 
We experienced some frustration using this method for the following reasons: (1) no one was there to answer the 
phone, (2) people would hang up on us, and (3) appropriate times to call are limited. Wethen chose to implement 
a traditional paper-and-pencil survey that was distributed to return customers in the retail shop. This method had 
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Case Study continued 

the highest return rate, which accounted for 50 percent of thesurveys. 

We entered the survey data into SPSS (a software package) to analyze the results. The research objectives assisted 
usin compiling the evaluation report for the business and for delivering our oral presentation totheclass. 

Step 5: Evaluation Project Report 

Based on thedata analysis, we felt overall that Appalachian Angler's customers were satisfied with theretail shop 
andthefishingguideservice. However, weidentified areasof concern when examiningtheeffectivenessof theweb 
page. For example, customers did not prefer shopping online, but ironically thedata suggested that most of the 
customerswould liketheoption of purchasing gear and bookingtrips online. In the I after stage of theevaluation 
process, we had to make recommendations to the company. We felt the customers' web page concerns needed 
to be addressed. Based on the identified customer concerns, we suggested that the company consider adding the 
option of booking fishing trips and purchasing gear online. 

According to class procedures, we had to reflect on and report our personal insights about the service-learning 
project. We reported that at ti mes we were very frustrated with using the various survey techniques ( i . e. , online, 
telephone, and paper-and-pencil). If we were to conduct another evaluation of a company, we would be more 
organized and allow more time to collect survey data. Finally, we commented on how thetrilogy of evaluation 
model assisted our research project by keeping usconnected to our research objectives. The model also guided us 
in making appropriate conclusion and recommendation statements to thecompany. 


by providing the students with an overview of the project, 
includingthegoals, objectives, and requirements to complete 
theproject. After thisinitial discussion, studentsareal lowed 
to establish their own evaluation teams (three students per 
team). According to Moorman and Arellano-Unruh (2002), 
allowing students to decide their own team members en- 
courages them to take ownership of the project, which is 
important to the project's success. 

During week three, each team selects an agency that 
they will evaluate. Students must make contact with the 
agency and together determine what the criteria will be for 
the evaluation. Through these negotiations with a supervi- 
sor of the agency, the team identifies the objectives of the 
evaluation project. 

By week four, teamsneed to solidify theevaluation criteria 
and objectives with theagency. They also need to solicit writ- 
ten permission from the agency to conduct the evaluation 
and submit this to the instructor. 

Step 2: The Proposal 

Duringthefifth week of the course, each team should submit 
a word-processed proposal to theinstructor for approval. This 
proposal should contain the following five specific compo- 
nents: (1) evaluation objectives; (2) background information 
on the agency, including contact person and their contact 
information; (3) method for conducting thesurvey (how they 
plan to collect the data); (4) project schedule for the entire 
semester; and (5) a brief literature review that supports their 
evaluation objectives or variables. Upon instructor approval, 
each team submits their proposal to theagency. 

Step 3: Survey Development 

During weeks six and seven, each team submits a survey 
instrumentto theinstructor. After the survey isrefined and 


graded according to agreed evaluation objectives of the 
agency, the evaluation team will then present their survey 
to th e cl ass an d th e i n structor for a verbal cri ti q ue. Th e peer- 
group critique isextremely beneficial to theevaluation team 
because it raises questions regarding both content validity 
and reliability of thesurvey. Attheend of week seven, each 
team presents the final draft of thesurvey to theagency for 
approval. Oncethesurvey has been approved, theteam can 
begin to collect data. 

Step 4: Data Collection 

This step in the process involves the collection of the data 
using the approved survey. During weeks eight through 
10, the team will work in the agency setting to collect the 
necessary data. Most data collection requires multipledays. 
After completing the data collection, the team focuses on 
interpretingthedataand creating the evaluation report. Dur- 
ing weeks 11 through 13, the students learn to use various 
computer programs to create graph sand slide presentations 
for their reports. 

Step 5: Presenting the Evaluation Project 
Report 

In thefinal step of theprocess, each team submits theirfinal 
written report to the instructor and presents their material 
to the class and the agency. This professional presentation 
lasts 20 to 25 minutes. One section of the presentation re- 
quires theteam to provide recommendation son theoverall 
evaluation design. Also, they must provide examples of the 
benefits and challenges of the process. During this period, 
students are amazed at how survey participants often make 
the same mistakes that are addressed in both lectures and 
readings. In addition, the presentation includesa question- 
and-answer period from the audience. This provides the 
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Table 2. Benefits and Challenges of Service-Learning Project 


Agency Benefits 

Theevaluation report assisted us 
with our marketing efforts. 

The project increased our under- 
stand i n g of wh i ch states atten d 
the Camp Fair. 

The enthusiasm and professional- 
ism of the students was a breath of 
fresh air to our agency. 

The economic impact evaluation 
summary allowed our association 
to lobby for hotel discounts for 
next year's tournament. 


Student Benefits 

Organizing the material helped me 
professionally. 

We gathered a lot of data and were 
able to apply it in a straightforward 
evaluation report. 

Learning the evaluation process and 
having it explained by my friends 
increased my understanding. 

I learned more than I ever expected 
about evaluation and surveys. This 
project put everything together. 


Professor Benefits 

Students wake up in this course and 
represent our school quite well. 

This project gives us a better visual 
presence in the community. 

This course promotes learning and 
benefits our local recreation providers 

Thisproject allows students to under- 
stand group cohesion and they get to 
network with practitioners. 


Agency Challenges 

Students did not have the exper- 
tise to evaluate what we wanted. 

Our image ison the lineand we 
want to make sure the students 
represent our agency in a profes- 
sional manner. 

We did not know what to evaluate. 
In the future we will focus on our 
program objectives and instructor 
assessments. 

Last year's group was much more 
prepared and did a better job. 


Student Challenges 

Doing survey research and ap- 
proaching participants was uncom- 
fortable. 

Meeting with other group members 
was al ways a ch al I en ge. 

Trying to learn thetheory and apply it 
at the same ti me was overwhel mi ng. 

Time management and data collec- 
tion was a big drawback for our group. 


Professor Challenges 

Preparing the students to meet all the 
challenges of an evaluation project. 

Motivating students to complete their 
assignment so they will understand 
all of the components of theevalua- 
tion project. 

Make sure the related assignments 
are graded and immediately returned 
so they can model the assignment in 
their evaluation project. 

Expecting too much from them or 
n arro wi n g th e scope of th e proj ect. 


teams with the opportunity to reflect on the significance of 
the overall project. 

Benefits and Challenges 

Comments about the benefits and challenges of this pro- 
cess— from students, instructors, and agency personnel— ap- 
pear in table 2. Service-learning projects are certainly chal- 
lenging; however, these challenges can benefit all involved. 
Clark (2002) suggested that when challenges are seen as 
opportunities for growth and learning, they are accepted 
and even welcomed. One major challenge during a semes- 
ter-long project isstudents' timecommitment. Studentswill 
need to meet outside of the classroom. However, this can 
be turned into a benefit, as it encourages students to learn 
time management and to work together effectively. It also 
enhances the students' professional growth and prepares 
them for real-world settings. 

Researchers have suggested that students should be re- 
minded that the project requires them to multi-task and to 
provide long-term effort and sustained commitment to the 
agency. Thisisdifficultforsomestudents, sincethey typically 
havelimited skills, meager knowledge, and littlecommitment 


to theagency (Estes, Wilson, & Toupence, 2001). 

Another foreseen challenge is the grading of the project. 
To reduce this challenge, the instructor should develop an 
objective assessment for both the written report assignment 
and the oral presentation. This is extremely important, be- 
cause the various agencies may have different criteria and 
evaluation objectives that are to be measured. 

Another benefit of the service-learning project examined 
in th i s arti cl e i s th at it assiststhe students in comprehending 
evaluation conceptsdueto itsfocuson several different learn- 
ing styles: auditory (listening to a brief lecture), participatory 
(becomingan active I earner), and retention (explaining what 
the data represents by using relevant terminology). Clark 
(2002) implied that the benefits outweigh thechallenges in 
service-learning and that every student receives something 
different from the experience. Students move from being 
dependent learners, to independent learners, and finally to 
interdependent learners. It is hoped that the students will 
comprehend that unity and teamwork, combined with the 
knowledge of others, is the only way to satisfy the needs 
of the community. This is an invaluable lesson that will 

Continues on page 54 
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resonate throughout their lives. A 
major factor associated with a posi- 
tive service-learning experience is 
the student's degree of interest in 
the subject matter. This allows the 
student to view education asa learn- 


ing experience, rather than as just 
the memorization of rote know- 
ledge (Astin, Vogel gesang, Ikeda, & 
Yee, 2000). 
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Ed i tori al 

Cont i nu e d from p a g e 5 

th e tr i ad to h i d e i n th e shadow 
of popu l ar ob e s i ty i ssu e s. I t i s 
i mportant for thos e work i ng w i th 


young f e ma le s, i nc l ud i ng par e nts, 
phys i ca l e ducators, and coach e s, 
to b e tra i n e d about e n e rgy n ee ds 
dur i ng i nt e ns e workouts and to 
und e rstand th e nutr i t i ona l r e qu i r e- 
m e nts for norma l growth and 
d e v el opm e nt dur i ng ado le sc e nc e 
(Go l d e n, 2002). I t a l so i s i mportant 
that phys i ca l e ducat i on t e ach e rs 
and coach e s d ee mphas i z e th e 
assoc i at i on b e tw ee n a le an body 
shap e and p e rformanc e and focus 
on e ducat i ng f e ma le s about sound 
nutr i t i ona l pr i nc i p le s for e nhanc - 
i ng ov e ra ll h e a l th (B e a l s, 2000). 
S i m il ar l y, i t i s i mportant to t e ach 
f e ma le s to work towards e nhanc - 
i ng p e rsona l h e a l th and w ell- b ei ng 
( e .g., str e ss manag e m e nt, mak i ng 
good food cho i c e s, e at i ng r e gu l ar 
m e a l s and h e a l thy snacks) i nst e ad 
of focus i ng on an arb i trary w ei ght 
goa l s e t by th e sport, a coach, 
par e nt, or fr ie nd (Manor e , 2002). 
F e ma le s can t ell i f th e y ar e mak i ng 
progr e ss by m e asur i ng pos i t i v e 
chang e s i n th ei r e n e rgy le v el s and 
p e rformanc e , not i ng a d e cr e as e i n 
i njur ie s and ill n e ss e s, th e r e turn of 
norma l m e nstruat i on, and g e n e ra l 
w ell- b ei ng (Manor e ). Ov e ra ll , i t 
i s e ss e nt i a l to e nsur e that young, 
act i v e f e ma le s consum e a h e a l thy 
d ie t and ma i nta i n th e e n e rgy 
n ee d e d i n ord e r to ma i nta i n a r e g - 
u l ar m e nstrua l cyc le for opt i ma l 
h e a l th and p e rformanc e . 
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